How can i use torch2trt with bert model——AttributeError: 'Tensor' object has no attribute '_trt'

[sunM123]

when i use bert model trained by pytorch , i have trouble like " has no attribute '_trt'", how can i change the model to trt type?

[sunM123]

Warning: Encountered known unsupported method torch.Tensor.hash
Warning: Encountered known unsupported method torch.Tensor.hash
Warning: Encountered known unsupported method torch.Tensor.to
Traceback (most recent call last):
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/torch2trt-0.3.0-py3.6.egg/torch2trt/torch2trt.py", line 552, in torch2trt
outputs = module(*inputs)
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/caomengdi/faq/rank_model/bert_classifier.py", line 75, in forward
bert_out = self.bert(token_ids, attention_mask=mask, token_type_ids=token_type_ids)
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 989, in forward
past_key_values_length=past_key_values_length,
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 201, in forward
position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/torch2trt-0.3.0-py3.6.egg/torch2trt/torch2trt.py", line 300, in wrapper
converter"converter"
File "/home/caomengdi/py3env/lib64/python3.6/site-packages/torch2trt-0.3.0-py3.6.egg/torch2trt/converters/getitem.py", line 30, in convert_tensor_getitem
input_trt = input._trt
AttributeError: 'Tensor' object has no attribute '_trt'

[JWLee89]

The error occurs because torch.Tensor.hash and torch.Tensor.to operations are layers / operations that are not supported by torch2trt.

You will have to do one of the following:

1. Replace torch.Tensor.hash and torch.Tensor.to with other operators
2. Write a tensorrt converter that converts the aforementioned layers to TensorRT layers

[jagadeeshi2i]

I am facing similar error with bert model. Working on this example.

Traceback (most recent call last):
  File "Download_Transformer_models.py", line 87, in <module>
    transformers_model_dowloader(mode,model_name, num_labels,do_lower_case, max_length, torchscript)
  File "Download_Transformer_models.py", line 68, in transformers_model_dowloader
    model_trt = torch2trt(model, (input_ids, attention_mask) )
  File "/opt/conda/lib/python3.6/site-packages/torch2trt-0.3.0-py3.6.egg/torch2trt/torch2trt.py", line 552, in torch2trt
    outputs = module(*inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 1533, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 955, in forward
    buffered_token_type_ids = self.embeddings.token_type_ids[:, :seq_length]
  File "/opt/conda/lib/python3.6/site-packages/torch2trt-0.3.0-py3.6.egg/torch2trt/torch2trt.py", line 300, in wrapper
    converter["converter"](ctx)
  File "/opt/conda/lib/python3.6/site-packages/torch2trt-0.3.0-py3.6.egg/torch2trt/converters/getitem.py", line 30, in convert_tensor_getitem
    input_trt = input._trt
AttributeError: 'Tensor' object has no attribute '_trt'

tensorrt=6.0.1.8
torch=1.9.0+cu111
transformers=4.10.2

[sunM123]

The error occurs because torch.Tensor.hash and torch.Tensor.to operations are layers / operations that are not supported by torch2trt.

You will have to do one of the following:

1. Replace torch.Tensor.hash and torch.Tensor.to with other operators
2. Write a tensorrt converter that converts the aforementioned layers to TensorRT layers

which operators can torch.Tensor.hash and torch.Tensor.to be changed to?

[JWLee89]

which operators can torch.Tensor.hash and torch.Tensor.to be changed to?

torch.Tensor.hash - I don't have much experience using the has function in PyTorch, so best option will be to write a conversion layer after fiddling with the function and getting accustomed the algorithm.

torch.Tensor.to - I am assuming that .to() is used to cast tensors to a different data type. If so, you can replace it explicitly to a casting operation such as .float(). Unfortunately, casting operations are not supported as of now, but I wrote custom converters and made a pull request here.

[JWLee89]

Coming back to this topic, are all the devices (such as input tensors, model) on the same gpu device or on multiple devices?
I recommend trying to place all tensors on GPU id=0.

torch2trt does not work well on multiple gpus