Chap 3: Fault about TransformerForSequenceClassification #144
Description
Information
The question or comment is about chapter:
- Introduction
- Text Classification
- Transformer Anatomy
- Multilingual Named Entity Recognition
- Text Generation
- Summarization
- Question Answering
- Making Transformers Efficient in Production
- Dealing with Few to No Labels
- Training Transformers from Scratch
- Future Directions
Question or comment
In TransformerForSequenceClassification, x = self.encoder(x)[:, 0, :] means [CLS] token is included in the inputs. However, in the beginning of this chapter, inputs is defined as tokenizer(text, return_tensors="pt", add_special_tokens=False), without special_tokens. Hence, the 0-th is "time", not "[CLS]".