[Bug]: Drop sentencepiece dependency

[tibbe]

Describe the bug

sentencepiece seems no longer maintained--the last release was Feb 19, 2024-- and doesn't build on Python 3.13 on several platforms/configurations (e.g. macOS):

• Particular failure I had: PIP INSTALL failing on OSX google/sentencepiece#1103
• Bug tracker has lots of issues with 3.13 build failures that have gone unaddressed: https://github.com/google/sentencepiece/issues

Since the package seems unmaintained, could flair drop the [sentencepiece] part of its transformers[sentencepiece] dependency so flair itself can build on macOS with Python 3.13?

To Reproduce

Build the latest sentencepiece (0.2.0) on e.g. macOS usng Python 3.13.

Expected behavior

flair builds on macOS with Python 3.13.

Logs and Stack traces

Screenshots

No response

Additional Context

No response

Environment

(P.S. The collect_env.py link above is wrong, it's missing the repo part of the URL.)

#### Versions:
##### Flair
0.15.0
##### Pytorch
2.5.1
##### Transformers
4.47.1
#### GPU
False

[stefan-it]

I think this should be possible:

SentencePiece directly is only used in the BytePairEmbeddings classhttps://github.com/flairNLP/flair/blob/d4ea3777998ba67bfbe6b6b8359e024dcf673c3e/flair/embeddings/token.py#L1368, via from sentencepiece import SentencePieceProcessor. We could remove that import and add a kind of lazy import check in the BytePairEmbeddings` class.

Using models within Transformers library: whenever no fast tokenizer is available for a model (from the Hugging Face Model Hub), then sentencepiece is necessary. However, a lot of famous models provide a fast tokenizer, so an error is thrown from Transformers library. That means, we could also remove the transformers[sentencepiece] dependency - as Transformers will throw an error - but only in corner cases.

But let's discuss that with @alanakbik :)