GGUF support

[zhaopengme]

I use Docker for deployment. I downloaded a 7B file, and the environment configuration file looks like this.

CRIA_SERVICE_NAME=cria
CRIA_HOST=0.0.0.0
CRIA_PORT=3000
CRIA_MODEL_ARCHITECTURE=llama

!/! Utilizado en docker-compose

CRIA_MODEL_PATH=/llama/llama-2-7b/consolidated.00.pth
CRIA_USE_GPU=true
CRIA_GPU_LAYERS=32
CRIA_ZIPKIN_ENDPOINT=http://zipkin-server:9411/api/v2/spans

[AmineDiro]

Hello,
This won't work because you are providing a '.pth' pytorch file. cria only works with ggml bin model. You'll need to download those.
You can find those on huggingface :
https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML

[UmutAlihan]

GGML type is being deprecated globally among all other repos, I think we need to update this repo so that it works well with GUFF as well

[AmineDiro]

Yes I do agree that a port to GUFF, I renamed this issue as a support for the new format.