Fix: Set TOKENIZERS_PARALLELISM to false

[Nsumithreddy]

Hi @kyleesays,

This PR is for Issue #2007.

I fixed the tokenizer warning by adding the os.environ["TOKENIZERS_PARALLELISM"] = "false" line to the src/llmcompressor/__init__.py file, like you suggested.

I couldn't get the warning to happen on my computer, but this fix should work.

Thanks.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Code Review

This pull request aims to fix a tokenizer warning by setting TOKENIZERS_PARALLELISM to false. While this is a valid workaround for the warning, setting an environment variable directly in the library's __init__.py is not ideal as it creates a global side-effect for any user importing the library. My review includes a suggestion to handle this in a less invasive way, improving the library's behavior as a dependency in other projects.

[kylesayrs]

Hi @Nsumithreddy, thanks for the diff!

I think that having LLM Compressor set environment variables is not good practice, as it has side effects for other tokenizer issues. Instead, I think it's important to figure out the source of these warnings. If a change needs to be made, we can include it in the example scripts with a quick explanation as to why these warnings happen and why the environment variable change is required.

[kylesayrs]

Closing because I don't think having LLM Compressor set 3rd party environment variables is the correct approach. Feel free to open another PR once we know what is triggering the warning and whether we should add anything to the example scripts