connection error

[marcorossi85]

I installed Newelle in 2 different pc (ubuntu distro) with flatpak,
In the settings, I chose to run local models. On one of these PCs, I rebuilded llama.cpp with hardware acceleration (by the way, I recommend improving your tutorials because some important steps for compiling llama.cpp are not explained; if you want, I can contribute by sending you the text to add). I downloaded two local models suitable for my hardware. On both PCs, I tried chatting with both models and always received a “connection error.”
On the PC where I recompiled llama.cpp with hardware acceleration, I tried to run the models from the terminal by launching them with the newly compiled llama.cpp binary, and they work from the terminal.
Can you help me understand where the problem is?
I understood that Newelle runs the local models downloaded with llama.cpp without any configuration, out-of-the-box. But in both installation it doesn't work.
Thank you so much.

[FrancescoCaracciolo]

Can you give terminal output when runnong with
flatpak run io.github.qwersyk.Newelle ?

[marcorossi85]

Unrecognized value "gl-disable". Try GDK_DEBUG=help
Unrecognized value "vulkan-disable". Try GDK_DEBUG=help
llama-server: /usr/lib/x86_64-linux-gnu/libcurl.so.4: no version information available (required by llama-server)
load_backend: loaded RPC backend from /app/bin/libggml-rpc.so
load_backend: loaded CPU backend from /app/bin/libggml-cpu-alderlake.so
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 7662 (56d2fed2b) with GNU 11.4.0 for Linux x86_64
system info: n_threads = 8, n_threads_batch = 8, total_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |

init: using 7 threads for HTTP server
start: binding port with default address family
main: loading model
srv load_model: loading model '/home/ensk320/.var/app/io.github.qwersyk.Newelle/config/models/custom_models/ministral-3-8b-reasoning.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: no devices with dedicated memory found
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.46 seconds
srv log_server_r: request: GET /v1/models 127.0.0.1 503
llama_model_loader: loaded meta data with 50 key-value pairs and 309 tensors from /home/ensk320/.var/app/io.github.qwersyk.Newelle/config/models/custom_models/ministral-3-8b-reasoning.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = mistral3
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = mistralai_Ministral 3 8B Reasoning 2512
llama_model_loader: - kv 3: general.version str = 2512
llama_model_loader: - kv 4: general.finetune str = Reasoning
llama_model_loader: - kv 5: general.basename str = mistralai_Ministral-3
llama_model_loader: - kv 6: general.size_label str = 8B
llama_model_loader: - kv 7: general.license str = apache-2.0
llama_model_loader: - kv 8: general.base_model.count u32 = 1
llama_model_loader: - kv 9: general.base_model.0.name str = Ministral 3 8B Base 2512
llama_model_loader: - kv 10: general.base_model.0.version str = 2512
llama_model_loader: - kv 11: general.base_model.0.organization str = Mistralai
llama_model_loader: - kv 12: general.base_model.0.repo_url str = https://huggingface.co/mistralai/Mini...
llama_model_loader: - kv 13: general.tags arr[str,1] = ["mistral-common"]
llama_model_loader: - kv 14: general.languages arr[str,11] = ["en", "fr", "es", "de", "it", "pt", ...
llama_model_loader: - kv 15: mistral3.block_count u32 = 34
llama_model_loader: - kv 16: mistral3.context_length u32 = 262144
llama_model_loader: - kv 17: mistral3.embedding_length u32 = 4096
llama_model_loader: - kv 18: mistral3.feed_forward_length u32 = 14336
llama_model_loader: - kv 19: mistral3.attention.head_count u32 = 32
llama_model_loader: - kv 20: mistral3.attention.head_count_kv u32 = 8
llama_model_loader: - kv 21: mistral3.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 22: mistral3.attention.key_length u32 = 128
llama_model_loader: - kv 23: mistral3.attention.value_length u32 = 128
llama_model_loader: - kv 24: mistral3.vocab_size u32 = 131072
llama_model_loader: - kv 25: mistral3.rope.dimension_count u32 = 128
llama_model_loader: - kv 26: mistral3.rope.scaling.type str = yarn
llama_model_loader: - kv 27: mistral3.rope.scaling.factor f32 = 16.000000
llama_model_loader: - kv 28: mistral3.rope.scaling.yarn_beta_fast f32 = 32.000000
llama_model_loader: - kv 29: mistral3.rope.scaling.yarn_beta_slow f32 = 1.000000
llama_model_loader: - kv 30: mistral3.rope.scaling.yarn_log_multiplier f32 = 1.000000
llama_model_loader: - kv 31: mistral3.rope.scaling.original_context_length u32 = 16384
llama_model_loader: - kv 32: mistral3.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 33: mistral3.attention.temperature_scale f32 = 0.100000
llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 35: tokenizer.ggml.pre str = tekken
llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,131072] = ["", "", "", "[INST]", "[...
llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,269443] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ �...
llama_model_loader: - kv 39: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 40: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 41: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 42: tokenizer.ggml.padding_token_id u32 = 11
llama_model_loader: - kv 43: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 44: tokenizer.ggml.add_sep_token bool = false
llama_model_loader: - kv 45: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 46: tokenizer.chat_template str = {#- Default system message if no syst...
llama_model_loader: - kv 47: tokenizer.ggml.add_space_prefix bool = false
llama_model_loader: - kv 48: general.quantization_version u32 = 2
llama_model_loader: - kv 49: general.file_type u32 = 15
llama_model_loader: - type f32: 69 tensors
llama_model_loader: - type q4_K: 205 tensors
llama_model_loader: - type q6_K: 35 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 4.83 GiB (4.89 BPW)
Error getting llamacpp models: Invalid port: 'None'
load: 0 unused tokens
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load: - 2 ('')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch = mistral3
print_info: vocab_only = 0
print_info: no_alloc = 0
print_info: n_ctx_train = 262144
print_info: n_embd = 4096
print_info: n_embd_inp = 4096
print_info: n_layer = 34
print_info: n_head = 32
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 0
print_info: is_swa_any = 0
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 4
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 14336
print_info: n_expert = 0
print_info: n_expert_used = 0
print_info: n_expert_groups = 0
print_info: n_group_used = 0
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = yarn
print_info: freq_base_train = 1000000.0
print_info: freq_scale_train = 0.0625
print_info: n_ctx_orig_yarn = 16384
print_info: rope_yarn_log_mul= 1.0000
print_info: rope_finetuned = unknown
print_info: model type = 8B
print_info: model params = 8.49 B
print_info: general.name = mistralai_Ministral 3 8B Reasoning 2512
print_info: vocab type = BPE
print_info: n_vocab = 131072
print_info: n_merges = 269443
print_info: BOS token = 1 ''
print_info: EOS token = 2 ''
print_info: UNK token = 0 ''
print_info: PAD token = 11 ''
print_info: LF token = 1010 'Ċ'
print_info: EOG token = 2 ''
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = true)
srv log_server_r: request: GET /v1/models 127.0.0.1 503
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
Error getting llamacpp models: Error code: 503 - {'error': {'message': 'Loading model', 'type': 'unavailable_error', 'code': 503}}
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
srv log_server_r: request: GET /v1/models 127.0.0.1 503
load_tensors: offloading output layer to GPU
load_tensors: offloading 33 repeating layers to GPU
load_tensors: offloaded 35/35 layers to GPU
load_tensors: CPU_Mapped model buffer size = 4918.55 MiB
load_tensors: CPU_REPACK model buffer size = 3404.25 MiB
......................srv log_server_r: request: GET /v1/models 127.0.0.1 503
..........srv log_server_r: request: GET /v1/models 127.0.0.1 503
..........srv log_server_r: request: GET /v1/models 127.0.0.1 503
..........srv log_server_r: request: GET /v1/models 127.0.0.1 503
.....srv log_server_r: request: GET /v1/models 127.0.0.1 503
.....srv log_server_r: request: GET /v1/models 127.0.0.1 503
....srv log_server_r: request: GET /v1/models 127.0.0.1 503
...srv log_server_r: request: GET /v1/models 127.0.0.1 503
....srv log_server_r: request: GET /v1/models 127.0.0.1 503
...srv log_server_r: request: GET /v1/models 127.0.0.1 503
....srv log_server_r: request: GET /v1/models 127.0.0.1 503
....srv log_server_r: request: GET /v1/models 127.0.0.1 503
..srv log_server_r: request: GET /v1/models 127.0.0.1 503
.srv log_server_r: request: GET /v1/models 127.0.0.1 503
.
common_init_result: added logit bias = -inf
llama_context: constructing llama_context
llama_context: setting new yarn_attn_factor = 1.0000 (mscale == 1.0, mscale_all_dim = 1.0)
llama_context: n_seq_max = 4
llama_context: n_ctx = 262144
llama_context: n_ctx_seq = 262144
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = auto
llama_context: kv_unified = true
llama_context: freq_base = 1000000.0
llama_context: freq_scale = 0.0625
llama_context: CPU output buffer size = 2.00 MiB
ggml_aligned_malloc: insufficient memory (attempted to allocate 34816.00 MB)
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 36507222016
alloc_tensor_range: failed to allocate CPU buffer of size 36507222016
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
common_init_result: failed to create context with model '/home/ensk320/.var/app/io.github.qwersyk.Newelle/config/models/custom_models/ministral-3-8b-reasoning.gguf'
common_init_from_params: failed to create context with model '/home/ensk320/.var/app/io.github.qwersyk.Newelle/config/models/custom_models/ministral-3-8b-reasoning.gguf'
['system', 'user']

[FrancescoCaracciolo]

You do not have memory to load the model, run
killall llama-server to kill zombie processes that might have been created

[marcorossi85]

No active processes found.

Furthermore, this insufficient memory issue seems strange to me: when I run the command

llama-cli -m modelname.gguf

the models were executed without any problems on the same computer. I also tested Jan on the same computer, and it loads and runs models that are not too big without any problems...

Francesco, what do these errors mean? Could they be related to this problem?

Error getting llamacpp models: Error code: 503 - {'error': {'message': 'Loading model', 'type': 'unavailable_error', 'code': 503}}

ggml_aligned_malloc: insufficient memory (attempted to allocate 34816.00 MB)
ggml_backend_cpu_buffer_type_alloc_buffer: failed to allocate buffer of size 36507222016
alloc_tensor_range: failed to allocate CPU buffer of size 36507222016
llama_init_from_model: failed to initialize the context: failed to allocate buffer for kv cache
common_init_result: failed to create context with model '/home/ensk320/.var/app/io.github.qwersyk.Newelle/config/models/custom_models/ministral-3-8b-reasoning.gguf'
common_init_from_params: failed to create context with model '/home/ensk320/.var/app/io.github.qwersyk.Newelle/config/models/custom_models/ministral-3-8b-reasoning.gguf'

[FrancescoCaracciolo]

Error code: 503 - {'error': {'message': 'Loading model', 'type': 'unavailable_error', 'code': 503}} This is fine, it's Newelle checking when the model is loaded

The other says that it can't allocate the model. Probably because you do not have enough vram/ram available? The issue might also be cause by some zombie Newelle processes as if Newelle crashes it is not able to kill them

[marcorossi85]

Thank you Francesco, sorry but I can't run in my pc, it need more resources.