Low Performance and Slow Execution on Town05 Short

[jhyoo7996]

Hello, I am currently testing the Interfuser model, and I have encountered several issues while running it on the Town05 short environment. Below are the details:

• Test Setup:
  ROUTE: leaderboard/data/validation_routes/routes_town05_short.xml
  SCENARIO: leaderboard/data/scenarios/town05_all_scenarios.json
  Model: http://43.159.60.142/s/p2CN

Despite running the model three times, the performance is lower than expected, as shown in the attached table.

• Issues:

1. Accuracy Issues
   The accuracy seems lower than anticipated. Is it possible that I should be using a different dataset for validation? If so, which one would you recommend for the Town05 short setup?
2. Execution Time
   The execution time is quite long. Is there any way to optimize the process or reduce the time spent during inference? Any suggestions would be appreciated.
3. Incorrect Recognition in Specific Scenarios
   In some special cases, the model seems to completely fail at recognizing or interpreting the data. I've attached an image of one such scenario. How can I resolve this? Is there a specific condition or setting that I may have missed?

• Environment:
  Ubuntu 20.04
  RTX 3090
  carla version carla/PythonAPI/carla/dist/carla-0.9.10-py3.7-linux-x86_64.egg

Looking forward to any insights or suggestions you might have for improving the model’s performance and reducing execution time.
Thank you!

[fuyz1231]

Hi,
Did you collect data for all towns under all weather conditions? Your accuracy is much better than mine. Did you train your model on an RTX 3090? How many epochs did you set, and how long did the training take?
Thanks

[jje526]

We only used the pretrained weights provided in this GitHub!

http://43.159.60.142/s/p2CN