[TabArena] Benchmarking BETA; Verifying our Implemenetation

[LennartPurucker]

Heyho,

We are working on benchmarking BETA-TabPFN on TabArena. I have implemented and benchmarked a first version of your model in this pull request: autogluon/tabrepo#172

Given the extensive refactoring required, please review it carefully and let me know if you find any issues.
Moreover, as we are pretty surprised by the performance, we are unsure if we used the correct HPs.

I used the default HPs from this repository (as I do not know how the tuned HPs were obtained and found no search space).
Please let me know if this is incorrect or if there's anything else I should be using.

I'm happy to discuss this further here or in the PR about how to maximize the performance of your model.
Feel free to test the implementation yourself or propose changes I should incorporate.

Once everyone is happy with the code, I can rerun the config and add its results to the public leaderboard.

Thank you!

[6sy666]

Thank you for your work on implementing and benchmarking BETA-TabPFN in TabArena! I’ll carefully review the pull request (#172) as soon as possible—I’m a bit tied up at the moment, but I’ll prioritize checking the implementation and hyperparameters to ensure everything aligns with expectations.
Thanks again for including BETA-TabPFN in your benchmarking efforts—I’m keen to collaborate and ensure the model’s performance is accurately represented.

[6sy666]

First, my sincere apologies for the delayed response. I've been juggling final exams, ongoing academic projects, my industry internship, and other obligations, which prevented me from reviewing the PR thoroughly earlier.

I've now reviewed your implementation (#172), and while the overall approach looks solid, I've identified one critical issue that needs attention:

Validation Data Handling (beta_model.py, ~L75)
In _fit(), both training and validation sets currently use X_train/y_train. This means the validation set isn't properly separated, which could lead to severe overfitting. The validation data (X_val/y_val) should be used explicitly for early stopping and hyperparameter tuning.

Inference Context Suggestion (talent_beta_model.py, ~L298)
For larger datasets, consider increasing the max_context_size during inference. Since GPU memory usage remains relatively low here, a larger context often improves performance without significant resource trade-offs.

Unfortunately, I'll be traveling for work shortly and won’t have bandwidth for deeper review at this moment. Once my schedule permits, I'll conduct further checks, explore potential improvements, and define a formal hyperparameter search space.

Thank you and your team again for driving this integration!

[LennartPurucker]

Thanks for reviewing the code! No pressure from my side, take all the time you need :)

I will fix the issues as soon as I find the time, and we can take it from there.
I am also traveling this week (ICML), so it might take a bit longer as I won't have too much bandwidth either.

[LennartPurucker]

New results:

Very nice :) !

[LennartPurucker]

On the TabPFN-TabICL data:

[6sy666]

Wow! Thank you for sharing the update so quickly!

[LennartPurucker]

Given that you closed the issue, do you believe your model is correctly represented on TabArena?

Some additional concerns could be: Are the hyperparameters correct, or do you want to run HPO over the model?

[6sy666]

Thanks for the reminder! I've reopened the issue. When time allows, I'll try to find a suitable search space for BETA to perform HPO.