COCO training error

[pjvazquez]

I'm trying to train the model with COCO Dataset following instructions, and I'm having this error

Traceback (most recent call last):
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/eager/context.py", line 2102, in execution_mode
yield
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 758, in _next_internal
output_shapes=self._flat_output_shapes)
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2610, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Reduction axis -1 is empty in shape [19248,0]
[[{{node parser/ArgMax}}]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 213, in
app.run(main)
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "train.py", line 149, in main
for image, labels in train_dataset:
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 736, in next
return self.next()
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 772, in next
return self._next_internal()
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 764, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/eager/context.py", line 2105, in execution_mode
executor_new.wait()
File "/home/pjvazquez/anaconda3/envs/tf20/lib/python3.7/site-packages/tensorflow/python/eager/executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Reduction axis -1 is empty in shape [19248,0]
[[{{node parser/ArgMax}}]]

Any other has?
Do you have a solution or a tip to fix it?

Thanks a lot

[bourrel]

I have the same error. Have you been able to fix it ?

[leohsuofnthu]

Hi Thanks for the question. I ve been stuck on my school work this year. I recently just start recheck and refactor the repo. I just finish rechecking the whole training pipeline this week. If you are still interested in using this repo. Would you mind trying to training again ? thx !! :))