Saving tf records issue in special case

[cjerzak]

Today I'm running several runs after making the changes we talked about regarding imputing project end dates. One of them failed with this error after initially appearing to have successfully created the tfrecord file:

Error in py_call_impl(callable, call_args$unnamed, call_args$named) :
  ValueError: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor.
Run `reticulate::py_last_error()` for details.
Calls: AnalyzeImageConfounding ... GetElementFromTfRecordAtIndices -> <Anonymous> -> py_call_impl
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Execution halted
Total Elapsed Time: 00:57:22 for ch_160, run tfrec_emb_collapsed, iter: 1000, time_approach: collapsed

Here's more context from the logfile:

[1] "At index 2550"
[1] "Done! Finalizing tfrecords...."
[1] "[2023-09-20 12:50:24] Finished creating tfrecord file: ./data/interim/tfrecords/ch_160_collapsed_imp3.tfrecord"
[1] "Initializing the tensorflow environment..."
[1] "Looking for Python modules tensorflow, gc..."
[1] "Establishing connection with tfrecord"
[1] "Temporarily re-setting the wd to ./data/interim/tfrecords"
2023-09-20 14:50:27.621375: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [1]
         [[{{node Placeholder/_0}}]]
2023-09-20 14:50:27.746835: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [1]
         [[{{node Placeholder/_0}}]]
[1] "Calibrating first moments for input data normalization..."
[1] "Bootstrap iteration 0 of 30 [embeddings model class]"
2023-09-20 14:50:42.648889: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [1]
         [[{{node Placeholder/_0}}]]
2023-09-20 14:50:42.657771: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [1]
         [[{{node Placeholder/_0}}]]
2023-09-20 14:50:42.721679: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [1]
         [[{{node Placeholder/_0}}]]
[1] "[2023-09-20 12:50:42] 0.00% done getting image/video embeddings"
2023-09-20 14:50:42.792172: I tensorflow/core/common_runtime/executor.cc:1197] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype string and shape [1]
         [[{{node Placeholder/_0}}]]
2023-09-20 14:50:56.281008: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:424] Loaded cuDNN version 8904
2023-09-20 14:51:07.279595: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
[1] "[2023-09-20 12:51:08] 1.96% done getting image/video embeddings"
[1] "[2023-09-20 12:51:08] 3.92% done getting image/video embeddings"
[1] "[2023-09-20 12:51:08] 5.88% done getting image/video embeddings"
[1] "[2023-09-20 12:51:09] 7.84% done getting image/video embeddings"
[1] "[2023-09-20 12:51:09] 9.80% done getting image/video embeddings"
[1] "[2023-09-20 12:51:10] 11.76% done getting image/video embeddings"
[1] "[2023-09-20 12:51:10] 13.72% done getting image/video embeddings"
[1] "[2023-09-20 12:51:11] 15.68% done getting image/video embeddings"
[1] "[2023-09-20 12:51:11] 17.64% done getting image/video embeddings"
[1] "[2023-09-20 12:51:12] 19.60% done getting image/video embeddings"
[1] "[2023-09-20 12:51:12] 21.56% done getting image/video embeddings"
[1] "[2023-09-20 12:51:12] 23.52% done getting image/video embeddings"
[1] "[2023-09-20 12:51:13] 25.48% done getting image/video embeddings"
[1] "[2023-09-20 12:51:13] 27.44% done getting image/video embeddings"
[1] "[2023-09-20 12:51:14] 29.40% done getting image/video embeddings"
[1] "[2023-09-20 12:51:14] 31.36% done getting image/video embeddings"
[1] "[2023-09-20 12:51:14] 33.32% done getting image/video embeddings"
[1] "[2023-09-20 12:51:15] 35.28% done getting image/video embeddings"
[1] "[2023-09-20 12:51:15] 37.24% done getting image/video embeddings"
[1] "[2023-09-20 12:51:16] 39.20% done getting image/video embeddings"
[1] "[2023-09-20 12:51:16] 41.16% done getting image/video embeddings"
[1] "[2023-09-20 12:51:16] 43.12% done getting image/video embeddings"
[1] "[2023-09-20 12:51:17] 45.08% done getting image/video embeddings"
[1] "[2023-09-20 12:51:17] 47.04% done getting image/video embeddings"
[1] "[2023-09-20 12:51:17] 49.00% done getting image/video embeddings"
[1] "[2023-09-20 12:51:18] 50.96% done getting image/video embeddings"
[1] "[2023-09-20 12:51:18] 52.92% done getting image/video embeddings"
[1] "[2023-09-20 12:51:19] 54.88% done getting image/video embeddings"
[1] "[2023-09-20 12:51:19] 56.84% done getting image/video embeddings"
[1] "[2023-09-20 12:51:20] 58.80% done getting image/video embeddings"
[1] "[2023-09-20 12:51:20] 60.76% done getting image/video embeddings"
[1] "[2023-09-20 12:51:20] 62.72% done getting image/video embeddings"
[1] "[2023-09-20 12:51:21] 64.68% done getting image/video embeddings"
[1] "[2023-09-20 12:51:21] 66.64% done getting image/video embeddings"
[1] "[2023-09-20 12:51:22] 68.60% done getting image/video embeddings"
[1] "[2023-09-20 12:51:22] 70.56% done getting image/video embeddings"
[1] "[2023-09-20 12:51:22] 72.52% done getting image/video embeddings"
[1] "[2023-09-20 12:51:23] 74.48% done getting image/video embeddings"
[1] "[2023-09-20 12:51:23] 76.44% done getting image/video embeddings"
[1] "[2023-09-20 12:51:23] 78.40% done getting image/video embeddings"
[1] "[2023-09-20 12:51:24] 80.36% done getting image/video embeddings"
[1] "[2023-09-20 12:51:24] 82.32% done getting image/video embeddings"
[1] "[2023-09-20 12:51:25] 84.28% done getting image/video embeddings"
[1] "[2023-09-20 12:51:25] 86.24% done getting image/video embeddings"
[1] "[2023-09-20 12:51:25] 88.20% done getting image/video embeddings"
[1] "[2023-09-20 12:51:26] 90.16% done getting image/video embeddings"
[1] "[2023-09-20 12:51:26] 92.12% done getting image/video embeddings"
[1] "[2023-09-20 12:51:26] 94.08% done getting image/video embeddings"
[1] "[2023-09-20 12:51:27] 96.04% done getting image/video embeddings"
[1] "[2023-09-20 12:51:27] 98.00% done getting image/video embeddings"
[1] "[2023-09-20 12:51:28] 99.96% done getting image/video embeddings"
Error in py_call_impl(callable, call_args$unnamed, call_args$named) :
  ValueError: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor.
Run `reticulate::py_last_error()` for details.
Calls: AnalyzeImageConfounding ... GetElementFromTfRecordAtIndices -> <Anonymous> -> py_call_impl
In addition: There were 50 or more warnings (use warnings() to see the first 50)
Execution halted
Total Elapsed Time: 00:57:22 for ch_160, run tfrec_emb_collapsed, iter: 1000, time_approach: collapsed

[cjerzak]

GetElementFromTfRecordAtIndices rebuilt; test to see if rebuild fixes this.