train.py gives tensorflow ValueError: No variables provided

[WPSanberg]

I try to train the KittiSeg net on my own data, so I have generated Kitti-like groundtruth masks, exported my DATA and RUN directories and created train/val txt files (and put those in KittiSeg.json. When I then run train.py, it starts and initialises the net, but it fails on training with the tensorflow ValueError 'no variables provided'. That does not sound data-set related or so. Do you know what should I update/provide/change to fix that?

(...)
2017-04-06 15:42:59,665 INFO Creating Summary for: score_pool3/biases
2017-04-06 15:42:59,704 INFO Creating Summary for: upscore32/up_filter
Traceback (most recent call last):
  File "train.py", line 131, in <module>
    tf.app.run()
  File "/home/shah/tensorflow/tensorflow_v1.0.0_py3.5_gpu/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "train.py", line 127, in main
    train.do_training(hypes)
  File "incl/tensorvision/train.py", line 377, in do_training
    tv_graph = core.build_training_graph(hypes, queue, modules)
  File "incl/tensorvision/core.py", line 100, in build_training_graph
    global_step, learning_rate)
  File "/home/shah/Coding/KittiSeg/hypes/../optimizer/generic_optimizer.py", line 91, in training
    global_step=global_step)
  File "/home/shah/tensorflow/tensorflow_v1.0.0_py3.5_gpu/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 380, in apply_gradients
    raise ValueError("No variables provided.")
ValueError: No variables provided.

possibly related things:

• I commented out line 123 (train.maybe_download_and_extract(hypes) in train.py because otherwise it would still download kitti;
• I got the same commentjson import error as issue When I run train.py, I got an import Error #28, so I had to import json instead of commentjson in train.py

Could that last one cause some values to be empty or is it something else?

[wenouyang]

Hi WPSanberg,

It seems to me that we are meeting the similar problem, i posted my question in the previous thread.

Wenouyang

[obendidi]

Hello , I don't know where the problem is in what you did ,
but I've trained the model on my own dataset and it did work , I'll tell what I did , maybe it will help ,
I mainly changed the input file , and set it up so that it loads the images directly from train and validation folders , after loading it resizes them to the same resolution as Kitti images , and then it gets the ground truth mask from the hypes , and use it to read the 'gt_images' and separate it into the different classes, the rest I didn't touch ... hope it helps

[MarvinTeichmann]

To me this looks like the tensor loss/total_loss is not well defined. Can you check, whether the loss function actually gets computed correctly?

[WPSanberg]

Hi Marvin,
It indeed seems to be about the loss, the error rises in tensorvisions def build_training_graph(hypes, queue, modules): at this segment:
with tf.name_scope("Loss"): losses = objective.loss(hypes, decoded_logits, labels)
but I am not sure how to trace it beyond that.

By the way, I get the same error message if I try to run the training process with the original configuration on the KITTI data.

[WPSanberg]

It seems to be python-related, I changed from python 3.5 to python 2.7 and the KITTI training works out of the box now.

[MarvinTeichmann]

Yeah, if I find some spare time, I will look into python 3.5. In theory it should work out of the box.

[dufengtong]

It because the return format of zip() is different between python2 and 3.
change the line grads_and_vars = zip(clipped_grads, tvars) in optimizer/generic_optimizer.py to grads_and_vars = list(zip(clipped_grads, tvars)), it works for me.

[KKayak]

@WPSanberg how to comment out line 123,train.maybe_download_and_extract(hypes),it should be what

[WPSanberg]

I already had the data and did not want the script to download it again, so I commented it out fully, without replacing it/changing it to something else. 2018-05-14 16:40 GMT+02:00 1464256670 <[email protected]>:

…

@WPSanberg <https://github.com/WPSanberg> how to comment out line 123,train.maybe_download_and_extract(hypes),it should be what — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZuCwbYVZlWlmz3zbC6lW5HSCBe4_AcJks5tyZdggaJpZM4M1s6E> .

[devimonica]

@WPSanberg how to comment out line 123,train.maybe_download_and_extract(hypes),it should be what

Nothing. Just go to Format and Comment out that region.