the latest llama cpp is unable to use the model suggested by the privateGPT main page

[thekit]

Hi All,

I got through installing the dependencies needed for windows 11 home
#230

but now the ingest.py script says my ggml model I downloaded from this github project is no good. Please help?

Successfully installed llama-cpp-python-0.1.50
PS C:\ai_experiments\privateGPT> python ./ingest.py
Loading documents from source_documents
Loaded 2 documents from source_documents
Split into 91 chunks of text (max. 500 tokens each)
llama.cpp: loading model from models/ggml-model-q4_0.bin
llama.cpp: can't use mmap because tensors are not aligned; convert to new format to avoid this
llama_model_load_internal: format = 'ggml' (old version with low tokenizer quality and no mmap support)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 1000
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 7B
error loading model: this format is no longer supported (see ggml-org/llama.cpp#1305)
llama_init_from_file: failed to load model
Traceback (most recent call last):
File "C:\ai_experiments\privateGPT\ingest.py", line 96, in
main()
File "C:\ai_experiments\privateGPT\ingest.py", line 87, in main
llama = LlamaCppEmbeddings(model_path=llama_embeddings_model, n_ctx=model_n_ctx)
File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCppEmbeddings
root
Could not load Llama model from path: models/ggml-model-q4_0.bin. Received error (type=value_error)
PS C:\ai_experiments\privateGPT>

[thekit]

I don't know how to requantize an existing model
ggml-org/llama.cpp#1305

so I am trying a q5 model from huggingface that says it has been requantized
https://www.reddit.com/r/LocalLLaMA/comments/13avdxb/gpt_for_all_13b_gpt4all13bsnoozygptq_is/
https://huggingface.co/TheBloke/gpt4-x-vicuna-13B-GGML/tree/main

the vicuna 5.1 seems to be ingesting my files now. my computer sounds like a jet engine lol

[thekit]

Solved! it is now able to make replies although not very coherently and not without program warnings

[thekit]

PERSIST_DIRECTORY=db
LLAMA_EMBEDDINGS_MODEL=models/gpt4-x-vicuna-13B.ggml.q5_1.bin
MODEL_TYPE=GPT4All
MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin
MODEL_N_CTX=1000

[raindrop2012]

Can we replace gpt4allcompletely with vicuna, as i can see there are gibberish reply with gpt4all

[davidtkeane]

I got the same error after python injest.py.

So i ran python3 injest.py an it worked for me.

I do not have the app working as I am getting another error, but I will download the huggingface and try this way also! Thank you.