Inquiry Regarding Fine-Tuning Pretrained SOTA Models in the Competition

[lavenderwfy]

Hello,
I am currently participating in the competition and have been exploring various approaches to enhance our model’s performance.
Given that the current SOTA models on the leaderboard have demonstrated strong results, I would like to inquire whether fine-tuning a pretrained version of some SOTA models with our own innovations would be considered a valid and compliant approach within the competition's rules.

Specifically, our approach involves:

• Utilizing the publicly available pretrained model of some SOTA methods.

• Introducing novel ideas and modifications to improve upon the existing architecture or training strategy.

• Fine-tuning the pretrained model with different training objectives to optimize performance.

Could you kindly confirm if this methodology aligns with the competition's regulations? Additionally, if there are any restrictions or clarifications regarding the use of pretrained models, I would greatly appreciate your guidance on how to proceed.

[janosh]

@lavenderwfy thanks for the question. that's definitely something we need to think about and come to a consensus on as a community. everyone is welcome weigh in.

here's a preliminary suggestion of how we could extend the PR template to specify requirements for fine-tuned submissions of existing models:

Additional Requirements for Fine-tuned Models

If your submission is a fine-tuned version of an existing model already on the leaderboard, please also check the following (if not, please delete this section):

• I have clearly indicated in the model name and description that this is a fine-tuned version of an existing model on the leaderboard
• I have included details of the fine-tuning process in the YAML metadata and readme.md
• My fine-tuned model shows significant improvement (>15% lower or higher depending on metric) in at least one metric compared to the original model
• My fine-tuned model does not show significant deterioration (>5%) in any metric compared to the original model
• I have included a comparison table or chart in the PR description showing the performance difference between the original and fine-tuned model
• I have included the fine-tuning script (finetune_<arch_name>.py) that was used to adapt the original model

maybe the deterioration criterion of 5% is too strict. but i also want to make sure, the leaderboard is not swamped with a dozen fine-tuned models that sacrifice overall performance to do well on a single metric

[lavenderwfy]

@janosh Thank you for the detailed response and for considering a structured approach to fine-tuned model submissions. I appreciate the effort to balance innovation with maintaining a meaningful leaderboard. Thanks again for your time and for fostering this discussion!