JavaScript/TypeScript Implementation of LLMLingua-2 (Experimental)

LLMLingua-2, Originally developed and implemented in Python by Microsoft, is a small-size yet powerful prompt compression method.

Efficient: Compresses context prompts with BERT sized models.
Accurate: Achieves high accuracy then other methods while requiring less computational resources.

llmlingua-2-js, ported by atjsh, is a pure JavaScript/TypeScript implementation of LLMLingua-2, designed to run in web browsers and Node.js environments.

Performance: Everything can be done in the browser. If your environment supports WebGPU, you can use it. Server-side processing is not required by default.
Correctness: The original logic will be ported to TypeScript as accurately as possible.

Try Demo Online (No Installation Required)

You can try it on the GitHub Pages without any installation.

The source code for demo is available in the examples directory. You can run it locally using the following command:

cd examples/react-vite-webgpu
yarn install

Learn More: Demo source code

Getting Started

This implementation depends on the following libraries:

Especially, the @huggingface/transformers library utilizes various computational optimizations to achieve high performance. Please consult if the running environment supports the minimum requirements from these libraries.

Installation

You can use the library by downloading the library from npm and importing it into your JavaScript/TypeScript project.

First, install the dependencies:

npm install @huggingface/transformers @tensorflow/tfjs js-tiktoken

Then, install the library:

npm install @atjsh/llmlingua-2

Model Selection

You can choose between models based on your needs.

XLM-RoBERTa
- Pros: High accuracy
- Cons: Slower, and slightly larger in size
- Public Model: atjsh/llmlingua-2-js-xlm-roberta-large-meetingbank
BERT
- Pros: Faster, smaller in size
- Cons: Lower accuracy compared to XLM-RoBERTa
- Public Model: Arcoldd/llmlingua4j-bert-base-onnx

Learn More about the performance of each model (actual performance may vary).

The model files will be downloaded automatically by default.

API Reference

For more details on how to use the library, please refer to the API reference documentation.

Testing

Unit tests are not available at the moment.

E2E tests are partially available in following directories:

src/e2e
examples/**

License

See LICENSE for details.

Credits

This software includes other software related under the following licenses:

LLMLingua (https://github.com/microsoft/LLMLingua), Copyright (c) Microsoft Corporation. (The original logic and implementation was licensed under the MIT license. For licensing, see: https://github.com/microsoft/LLMLingua/blob/main/LICENSE )

License

atjsh/llmlingua-2-js

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 3

Uh oh!

Languages