AutoML returning NoneType for model after tuning with small time budget

[OliverSchacht]

Hello,
I'm not sure if this is an issue or if it is me doing something wrong so I rather open a discussion.
I'm experimenting with parameter tuning by FLAML in different time budgets.
I have the problem, that for very small time budgets (<1s) FLAML AutoML sometimes, but inconsistently, returns a NoneType after fitting to the data. My code looks somehow like this

from flaml import AutoML
automl_y = AutoML()
automl_y.fit(X, y, task="regression", time_budget=0.7, metric="mse", verbose=False)
ml_tuned = automl_y.model.estimator

with X being a array shaped (1000,200). I get the following error:

---> 21 ml_tuned = automl_y.model.estimator
AttributeError: 'NoneType' object has no attribute 'estimator'

When I rerun the experiment with the same data, it works, but then stops later at another repetition again. Of course, an easy work-around is to catch the error and re-run the experiment, still I wonder, how this can happen, because from what I understood in the docs, it should return rather a default estimator, than a NoneType object? Can someone explain what happens?
Thank you very much,
Oliver

[sonichi]

When the time budget is too small, it can happen that no estimator is trained at all within the budget. Due to variance of the execution time for the code before a training starts, in some run you have enough time left to train one model and in some other run you don't.

[OliverSchacht]

We are investigating the interaction between hyperparameter tuning of the first stage regressions and the quality of causal inference in double machine learning to give practical insights on how to tune here.
We use the package DoubleML for inference on an average treatment effect parameter in a partially linear regression model with some nuisance that needs to be estimated.
The methodology is robust against small overfitting bias, however untuned or underfitting estimators have an unknown influence. Therefore we start with very little tuning budget and increase tuning time subsequently, to measure the MSE for the prediction problems as well as for the causal inference.
It looks a bit like this (with a loop over different t):

from flaml import AutoML
import doubleml as dml
from doubleml.datasets import make_plr_CCDDHNR2018

df = make_plr_CCDDHNR2018(return_type="DataFrame")

y, d, X = df["y"], df["d"], df.drop(columns=["y","d"])

automl_y = AutoML()
automl_y.fit(X, y, task="regression", time_budget=t)
automl_d = AutoML()
automl_d.fit(X, d, task="regression", time_budget=t)
ml_g = automl_y.model.estimator
ml_m = automl_d.model.estimator

dml_automl = dml.DoubleMLPLR(dml.DoubleMLData(df,y_col='y',d_cols='d'), ml_g = ml_g, ml_m = ml_m, n_folds = 4)
dml_automl.fit()

[sonichi]

Thanks for sharing. Are all the features in the df numeric? If not, flaml might perform some preprocessing of the data. The preprocessing is not contained in ml_g or ml_m, which could cause an issue.

[OliverSchacht]

Thanks for your feedback. All features are numeric, thus I think it should work this way.

[sonichi]

Yes that should work. Are you from academia or industry? Would you like to keep in touch on discord or gitter?

[OliverSchacht]

@sonichi : I sent you a mail with more details, I'm happy to get into exchange.