segmentation and depth ds #8

BobrG · 2020-08-19T18:05:16Z

No description provided.

rakhimovv · 2020-08-20T01:07:17Z

sharpf/utils/abc_utils/hdf5/dataset.py

@@ -133,3 +133,117 @@ def __getitem__(self, index):
            file_index_to_unload = np.random.choice(loaded_file_indexes)
            self.files[file_index_to_unload].unload()
        return item
+
+class DepthDataset(LotsOfHdf5Files):


what is the point of inheritance if you redefine everything? almost all logic working with hdf5 I suppose is already well implemented inside LotsOfHdf5Files class by @artonson. I suppose DepthDataset should mostly concentrate on data and target preprocessing without diving into hdf5 reading logic

rakhimovv · 2020-08-20T01:09:59Z

sharpf/utils/abc_utils/hdf5/dataset.py

+
+    def __getitem__(self, index):
+
+        data, target = self._getdata(index)


here it is better to do something like this

Suggested change

data, target = self._getdata(index)

data = super().__getitem__(index)

rakhimovv · 2020-08-20T01:12:30Z

sharpf/modeling/meta_arch/depth_segmentator.py

+            rmse_sum += output['rmse_sum']
+            size += output['batch_size']
+        mean_rmse = gather_sum(rmse_sum) / gather_sum(size)
+        logs = {f'{prefix}_mean_rmse': mean_rmse}


why I see regression metrics inside Segmentation PL module

rakhimovv · 2020-08-20T01:15:12Z

sharpf/utils/abc_utils/hdf5/dataset.py

+            target = torch.FloatTensor(dist_mask)
+
+        data = torch.FloatTensor(data).unsqueeze(0)
+        data = torch.cat([data, data, data], dim=0)


what is this? data = torch.cat([data, data, data], dim=0)

in my version of unet input had to have 3 channels. Should I remove this and keep one channel for input data?

remove. Unet in our reference implementation https://github.com/qubvel/segmentation_models.pytorch/blob/master/segmentation_models_pytorch/unet/model.py#L52 allows to setup in_channels param.

rakhimovv · 2020-08-20T01:18:48Z

sharpf/utils/abc_utils/hdf5/dataset.py

+        mask_1 = (np.copy(data) != 0.0).astype(float)  # mask for object
+        mask_2 = np.where(data == 0)  # mask for background
+
+        data = self.quantile_normalize(data)
+        data = self.standartize(data)
+
+        dist_new = np.copy(target)
+        dist_mask = dist_new * mask_1  # select object points
+        dist_mask[mask_2] = 1.0  # background points has max distance to sharp features
+        close_to_sharp = np.array((dist_mask != np.nan) & (dist_mask < 1.)).astype(float)


I suppose some of these preprocessing steps are task-specific and should be stated inside if closure

then I suppose it's better to add a normalisation parameter to cfg file and dataset init. This parameter would be a list of specific preprocessing methods to apply

Yes, could be a possible solution

rakhimovv · 2020-08-20T01:19:47Z

sharpf/utils/abc_utils/hdf5/dataset.py

+        data = torch.FloatTensor(data).unsqueeze(0)
+        data = torch.cat([data, data, data], dim=0)
+
+        return data, target


The better idea would be to return a dictionary instead of a tuple, so there is no ambiguity then what is a target

rakhimovv · 2020-08-20T01:20:58Z

The good point is also to try running code in the first place before committing

BobrG · 2020-08-20T11:53:18Z

@rakhimovv please see new changes though I haven't run code yet

rakhimovv · 2020-08-20T11:59:13Z

sharpf/modeling/meta_arch/depth_segmentator.py

+from torch.utils.data import DataLoader
+
+from sharpf.utils.comm import get_batch_size
+from sharpf.utils.losses import balanced_accuracy


I haven't found the implementation of balanced_accuracy

rakhimovv · 2020-08-20T12:01:16Z

sharpf/modeling/meta_arch/depth_segmentator.py

-        mean_rmse = gather_sum(rmse_sum) / gather_sum(size)
-        logs = {f'{prefix}_mean_rmse': mean_rmse}
-        return {f'{prefix}_mean_rmse': mean_rmse, 'log': logs}
+        mean_metric = gather_sum(metric_sum) / gather_sum(size)


I think the balanced accuracy needs a bit different gathering logic across batches

rakhimovv · 2020-08-20T12:03:27Z

Also, I do not see any changes in DepthDataset. Maybe you forgot to git add. Please first run code without crashes, then commit. For faster debugging you can set command line arg trainer.fast_dev_run=true

BobrG · 2020-08-20T12:11:02Z

added depth ds changes

rakhimovv · 2020-08-20T12:13:40Z

sharpf/utils/abc_utils/hdf5/dataset.py


-        return data, target
+        return {'data': data, 'target': target}


I meant something more informative :) like {'is_sharp_mask': ..., 'image': ...} or {'sharpness': ..., 'image': ...}. So the keys inside dict represent the task

so then keys should differ for each task? what's the purpose?

or they should be similar for each task in depth dataset, just have names which describe what kind of data is this?

for the sake of readability

and also for example, if you want to add later several new additional targets, it would be much easier to add them just like new keys: {'image': ..., 'sharpness': ..., 'normals': ..., 'whatever else': ...}

also, for segmentation task for ex. I have a binary close-to-sharp target and for regression a distance field, then there should be several returns for each task or same names

{'image': ..., 'distance_field': ..., 'close_to_sharp_mask': ...}

rakhimovv · 2020-08-21T11:47:24Z

train_net.py

@@ -36,7 +38,7 @@ def main(cfg: DictConfig):
    log.info(f"Original working directory: {hydra.utils.get_original_cwd()}")
    seed_everything(cfg.seed)

-    model = instantiate(cfg.meta_arch, cfg=cfg)
+    model = DepthRegressor(cfg)


train_net.py should be model-agnostic as it was

rakhimovv · 2020-08-21T11:48:09Z

sharpf/modeling/meta_arch/depth_segmentator.py

        points, distances = batch['image'], batch['distances']
        points = points.unsqueeze(1) if points.dim() == 3 else points
        preds = self.forward(points)
+        print(preds.shape, distances.shape)


do not include please debugging output into the commit

BobrG · 2020-08-21T12:23:11Z

@rakhimovv are there any other comments on my last commit?

rakhimovv · 2020-08-21T12:31:44Z

I still do not see balanced_accuracy implementation. You can google for an example of different segmentation metrics, how they were implemented by PL community. Maybe it was already implemented by somebody. Please make sure that it works in a multi-gpu regime. Check the docs about that here.

Also, check the last changes in pl_hydra branch and merge changes from pl_hydra to segmentation branch (not in the reverse direction, I will do it myself after approving this PR). The major change is due to facebookresearch/hydra#882, i.e. now we use _target_ instead of target and no more params key when instantiating new objects. You can see actual documentation here

BobrG · 2020-08-21T12:46:24Z

@rakhimovv please have a look at balanced accuracy
also, do you still have concerns about _shared_eval_step and _shared_eval_epoch_end from depth_segmentator.py ? should I reconsider the logic of metric calculation?

rakhimovv · 2020-08-21T12:54:04Z

@BobrG as it works now, you calculate the balanced accuracy on each "sub" batch on each gpu and then average it. Instead, you should output predictions and corresponding targets inside eval step, then gather all predictions and targets on epoch end and finally then calculate the metric. The gathering is necessary because otherwise, you skip data. Please read docs, I attached links above in the previous comment.

… fixed cfgs

BobrG · 2020-08-24T07:50:37Z

@rakhimovv can you please have a look at my last commit?

rakhimovv · 2020-08-24T13:17:56Z

Good job @BobrG. I will refactor and polish some minor things myself and after that will merge

segmentation and depth ds

eef7ba4

BobrG requested a review from rakhimovv August 19, 2020 18:05

rakhimovv reviewed Aug 20, 2020

View reviewed changes

corrected depth ds inheritance, metrics in segmentation model

bec69a8

rakhimovv reviewed Aug 20, 2020

View reviewed changes

corrected depth ds inheritance

bba6ff1

rakhimovv reviewed Aug 20, 2020

View reviewed changes

added balanced accuracy and ~after run~ fixes

5cb47c9

rakhimovv reviewed Aug 21, 2020

View reviewed changes

added balanced accuracy for real

92b454b

g.bobrovskih added 3 commits August 21, 2020 16:42

small fixes

a75cb74

commit before merge

bef0a33

changed balanced accuracy computation, changed dict keys in depth ds,…

c87b3c5

… fixed cfgs

Merge branch 'pl_hydra' into segmentation

21f4417

rakhimovv added 5 commits August 25, 2020 00:05

delete garbage

07c1a1e

add more distributed tools

56e1d2e

fix metric calculation

4b45ef4

fix typos

56d07cb

fix dimensions

aba17cc

rakhimovv merged commit 221d76a into pl_hydra Aug 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

segmentation and depth ds #8

segmentation and depth ds #8

BobrG commented Aug 19, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

BobrG Aug 20, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020

BobrG Aug 20, 2020

rakhimovv Aug 20, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv commented Aug 20, 2020

BobrG commented Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020

rakhimovv commented Aug 20, 2020 •

edited

Loading

BobrG commented Aug 20, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

BobrG Aug 20, 2020

BobrG Aug 20, 2020

rakhimovv Aug 20, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

BobrG Aug 20, 2020

rakhimovv Aug 20, 2020

rakhimovv Aug 21, 2020

rakhimovv Aug 21, 2020

BobrG commented Aug 21, 2020

rakhimovv commented Aug 21, 2020 •

edited

Loading

BobrG commented Aug 21, 2020

rakhimovv commented Aug 21, 2020 •

edited

Loading

BobrG commented Aug 24, 2020

rakhimovv commented Aug 24, 2020


		def __getitem__(self, index):

		data, target = self._getdata(index)

	data, target = self._getdata(index)
	data = super().__getitem__(index)

segmentation and depth ds #8

segmentation and depth ds #8

Conversation

BobrG commented Aug 19, 2020

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

rakhimovv commented Aug 20, 2020

BobrG commented Aug 20, 2020 • edited Loading

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhimovv commented Aug 20, 2020 • edited Loading

BobrG commented Aug 20, 2020

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhimovv Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BobrG commented Aug 21, 2020

rakhimovv commented Aug 21, 2020 • edited Loading

BobrG commented Aug 21, 2020

rakhimovv commented Aug 21, 2020 • edited Loading

BobrG commented Aug 24, 2020

rakhimovv commented Aug 24, 2020

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

BobrG commented Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv commented Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv Aug 20, 2020 •

edited

Loading

rakhimovv commented Aug 21, 2020 •

edited

Loading

rakhimovv commented Aug 21, 2020 •

edited

Loading