AttributeError: 'CallbackHandler' object has no attribute 'tokenizer' when using SetFitTrainer in zero shot cold-start notebook

[pavithranair]

Bug description

Hi! I'm running the notebook 04-zero-shot-cold-start.ipynb from the small-text repo. During execution, when calling:

clf.fit(cold_start_dataset, setfit_train_kwargs=setfit_train_kwargs)

I encounter the following error:

AttributeError: 'CallbackHandler' object has no attribute 'tokenizer'

Any assistance resolving this would be greatly appreciated.

Environment:

Python version: 3.11.13
small-text version: small-text[transformers]==2.0.0dev2
small-text integrations (e.g., transformers): small-text[transformers]==2.0.0dev2
PyTorch version (if applicable): N/A

Installation (pip, conda, or from source): pip
CUDA version (if applicable): N/A

Additional information

---

AttributeError Traceback (most recent call last)
/tmp/ipython-input-3480217872.py in <cell line: 0>()
37
38
---> 39 active_learner, train = initialize_with_coldstart()

8 frames
/tmp/ipython-input-3480217872.py in initialize_with_coldstart()
29
30 clf = clf_factory.new()
---> 31 clf.fit(cold_start_dataset, setfit_train_kwargs=setfit_train_kwargs)
32
33 active_learner.initialize(clf)

/usr/local/lib/python3.11/dist-packages/small_text/integrations/transformers/classifiers/setfit.py in fit(self, train_set, validation_set, setfit_train_kwargs)
265
266 self.model.model_body.to(self.device)
--> 267 return self._fit(sub_train, sub_valid, setfit_train_kwargs)
268
269 def _get_train_and_valid_sets(self, x_train, y_train, x_valid, y_valid):

/usr/local/lib/python3.11/dist-packages/small_text/integrations/transformers/classifiers/setfit.py in _fit(self, sub_train, sub_valid, setfit_train_kwargs)
280 def _fit(self, sub_train, sub_valid, setfit_train_kwargs):
281 seed = np.random.randint(np.iinfo(np.uint32).max, dtype=np.uint32).item()
--> 282 trainer = SetFitTrainer(
283 self.model,
284 sub_train,

/usr/local/lib/python3.11/dist-packages/setfit/trainer.py in init(self, model, train_dataset, eval_dataset, model_init, metric, metric_kwargs, loss_class, num_iterations, num_epochs, learning_rate, batch_size, seed, column_mapping, use_amp, warmup_proportion, distance_metric, margin, samples_per_label)
871 loss=loss_class,
872 )
--> 873 super().init(
874 model=model,
875 args=args,

/usr/local/lib/python3.11/dist-packages/setfit/trainer.py in init(self, model, args, train_dataset, eval_dataset, model_init, metric, metric_kwargs, callbacks, column_mapping)
326
327 callbacks = callbacks + [ModelCardCallback(self)] if callbacks else [ModelCardCallback(self)]
--> 328 self.st_trainer = BCSentenceTransformersTrainer(
329 setfit_model=model,
330 setfit_args=self.args,

/usr/local/lib/python3.11/dist-packages/setfit/trainer.py in init(self, setfit_model, setfit_args, callbacks, **kwargs)
92 for callback in callbacks:
93 self.add_callback(callback)
---> 94 self.callback_handler.on_init_end(self.args, self.state, self.control)
95
96 @Property

/usr/local/lib/python3.11/dist-packages/transformers/trainer_callback.py in on_init_end(self, args, state, control)
500
501 def on_init_end(self, args: TrainingArguments, state: TrainerState, control: TrainerControl):
--> 502 return self.call_event("on_init_end", args, state, control)
503
504 def on_train_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl):

/usr/local/lib/python3.11/dist-packages/setfit/trainer.py in (*args, **kwargs)
85 return control
86
---> 87 self.callback_handler.call_event = lambda *args, **kwargs: overwritten_call_event(
88 self.callback_handler, *args, **kwargs
89 )

/usr/local/lib/python3.11/dist-packages/setfit/trainer.py in overwritten_call_event(self, event, args, state, control, **kwargs)
73 st_model=self.model,
74 st_args=args,
---> 75 tokenizer=self.tokenizer,
76 optimizer=self.optimizer,
77 lr_scheduler=self.lr_scheduler,

AttributeError: 'CallbackHandler' object has no attribute 'tokenizer'

[pavithranair]

Update: On switching to transformers version 4.44.0 I was able to resolve this error, but now I am encountering

TypeError: SentenceTransformerModelCardCallback.on_init_end() missing 1 required positional argument: 'trainer'

while running the same code clf.fit(cold_start_dataset, setfit_train_kwargs=setfit_train_kwargs)

Stack Trace

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[/tmp/ipython-input-3480217872.py](https://localhost:8080/#) in <cell line: 0>()
     37 
     38 
---> 39 active_learner, train = initialize_with_coldstart()

8 frames
[/tmp/ipython-input-3480217872.py](https://localhost:8080/#) in initialize_with_coldstart()
     29 
     30     clf = clf_factory.new()
---> 31     clf.fit(cold_start_dataset, setfit_train_kwargs=setfit_train_kwargs)
     32 
     33     active_learner.initialize(clf)

[/usr/local/lib/python3.11/dist-packages/small_text/integrations/transformers/classifiers/setfit.py](https://localhost:8080/#) in fit(self, train_set, validation_set, setfit_train_kwargs)
    265 
    266         self.model.model_body.to(self.device)
--> 267         return self._fit(sub_train, sub_valid, setfit_train_kwargs)
    268 
    269     def _get_train_and_valid_sets(self, x_train, y_train, x_valid, y_valid):

[/usr/local/lib/python3.11/dist-packages/small_text/integrations/transformers/classifiers/setfit.py](https://localhost:8080/#) in _fit(self, sub_train, sub_valid, setfit_train_kwargs)
    280     def _fit(self, sub_train, sub_valid, setfit_train_kwargs):
    281         seed = np.random.randint(np.iinfo(np.uint32).max, dtype=np.uint32).item()
--> 282         trainer = SetFitTrainer(
    283             self.model,
    284             sub_train,

[/usr/local/lib/python3.11/dist-packages/setfit/trainer.py](https://localhost:8080/#) in __init__(self, model, train_dataset, eval_dataset, model_init, metric, metric_kwargs, loss_class, num_iterations, num_epochs, learning_rate, batch_size, seed, column_mapping, use_amp, warmup_proportion, distance_metric, margin, samples_per_label)
    871             loss=loss_class,
    872         )
--> 873         super().__init__(
    874             model=model,
    875             args=args,

[/usr/local/lib/python3.11/dist-packages/setfit/trainer.py](https://localhost:8080/#) in __init__(self, model, args, train_dataset, eval_dataset, model_init, metric, metric_kwargs, callbacks, column_mapping)
    326 
    327         callbacks = callbacks + [ModelCardCallback(self)] if callbacks else [ModelCardCallback(self)]
--> 328         self.st_trainer = BCSentenceTransformersTrainer(
    329             setfit_model=model,
    330             setfit_args=self.args,

[/usr/local/lib/python3.11/dist-packages/setfit/trainer.py](https://localhost:8080/#) in __init__(self, setfit_model, setfit_args, callbacks, **kwargs)
     92         for callback in callbacks:
     93             self.add_callback(callback)
---> 94         self.callback_handler.on_init_end(self.args, self.state, self.control)
     95 
     96     @property

[/usr/local/lib/python3.11/dist-packages/transformers/trainer_callback.py](https://localhost:8080/#) in on_init_end(self, args, state, control)
    454 
    455     def on_init_end(self, args: TrainingArguments, state: TrainerState, control: TrainerControl):
--> 456         return self.call_event("on_init_end", args, state, control)
    457 
    458     def on_train_begin(self, args: TrainingArguments, state: TrainerState, control: TrainerControl):

[/usr/local/lib/python3.11/dist-packages/setfit/trainer.py](https://localhost:8080/#) in <lambda>(*args, **kwargs)
     85             return control
     86 
---> 87         self.callback_handler.call_event = lambda *args, **kwargs: overwritten_call_event(
     88             self.callback_handler, *args, **kwargs
     89         )

[/usr/local/lib/python3.11/dist-packages/setfit/trainer.py](https://localhost:8080/#) in overwritten_call_event(self, event, args, state, control, **kwargs)
     66         def overwritten_call_event(self, event, args, state, control, **kwargs):
     67             for callback in self.callbacks:
---> 68                 result = getattr(callback, event)(
     69                     self.setfit_args,
     70                     state,

TypeError: SentenceTransformerModelCardCallback.on_init_end() missing 1 required positional argument: 'trainer'

[chschroeder]

Hi @pavithranair, thank you for reporting this issue.

Which versions of sentence-transformers and setfit are you using?

[pavithranair]

I am using setfit version 1.1.0 and sentence-transformers version 5.0.0.

[chschroeder]

Managed to get it working. In the first notebook cell go to the line:

%pip install setfit==1.1.0

and replace it with:

%pip install setfit==1.1.2 "sentence-transformers<5.0.0" "transformers==4.44.0" "datasets<4.0.0"
%env WANDB_DISABLED=true

Sorry for the inconvenience! Unfortunately, this kind of hassle can happen when integrating with other software. A detailed summary is written below.

---

Summary: There were multiple issues:

• New major versions for sentence-transformers and datasets.
• The error with the model callback was solved by a more recent setfit version
  (Address deprecated ModelCardCallback from sentence_transformers huggingface/setfit#603)
• As you correctly noticed, there was also a problem with transformers>4.44.0 which I did not investigate yet.
• WANDB_DISABLED=true is required, or otherwise setfit automatically tries to report to wandb which triggers a notebook input prompting for the API key. Might not be a problem with the next small-text version, where I use the non-deprecated setfit trainer.

I will try to fix this asap. Overall I am working on a setup that better tests against multiple versions of these dependencies. It's a lot of work though to do this properly, but it is on the horizon.

[pavithranair]

Thank you for the fix and the explanation, this worked for me. Appreciate the effort you put into maintaining compatibility across all these moving parts, looking forward to the next release!

[chschroeder]

Update: With v2.0.0.dev3 this issue is not fully solved, but has slightly improved.

%pip install "datasets<4.0.0"
%env WANDB_DISABLED=true

Controlling the datasets version and setting WANDB_DISABLED is sufficient to run the notebook successfully. The former should be solvable from my side, and for the latter I am hoping that this will be improved in the setfit repo (see huggingface/setfit#615).