feat: Add optional prompt processing progress streaming

[baonudesifeizhai]

• Add include_prompt_progress parameter to slot_params (default: false)
• Extend server_task_result_cmpl_partial with progress fields
• Implement send_progress_response() function with 1% progress intervals
• Add progress response in prompt processing loop
• Update JSON response to include prompt_processing field when requested
• Add comprehensive documentation to README.md
• Ensure full backward compatibility with existing clients

Closes #14685

Make sure to read the contributing guidelines before submitting a PR

[BradHutchings]

Is there a chance this could get approved? If it's not a welcome addition, I'll put it in the mmojo-server fork. Being able to display evaluating progress is a must for servers running on slow CPU, e.g. Raspberry Pi 5.

[ggerganov]

We can use a more compact send_progress, instead of include_prompt_progress. Otherwise seems a good change.

[ggerganov]

Suggested change

	params.include_prompt_progress = json_value(data, "include_prompt_progress", false);
	params.send_progress = json_value(data, "send_progress", false);

[BradHutchings]

Thanks, Georgi! And for all you do with llama.cpp. Nobody says thank you enough!

[ngxson]

earlier in the code, we already had a param called return_tokens so maybe return_progress is a better naming

[ngxson]

Is there a chance this could get approved?

There are multiple PRs already open for this particular feature, it's very hard for maintainers to keep track. It's better to look at the list of opening PR before working on a feature.

[ngxson]

At least one test case is also required for this feature, maybe with a long prompt and a small batch size so we can clear see the effect.

[BradHutchings]

Hold up on this a moment. I've been testing this implementation today.

        // Send progress if:
        // 1. This is the first progress update (last_progress == -1)
        // 2. Progress increased by at least 1% or processed at least 10 tokens
        // 3. We've completed processing (current_progress >= 1.0)
        bool should_send = (last_progress < 0.0f) || 
                          (current_progress - last_progress >= 0.01f) || 
                          (current_progress >= 1.0f && last_progress < 1.0f);

        if (!should_send) {
            return;
        }

This logic should be eliminated. We should send progress only when a batch is complete. Otherwise, a bunch of progress messages blast the client as the batch completes. Most of the time spent processing a batch is not spent in the steps that emit this progress. It's spent, for example, here, as show with LLAMA_PROFIL=1 and looking at the profile data with gprof:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 77.45    312.58   312.58     9920    31.51    31.51  void (anonymous namespace)::tinyBLAS_Q0_AVX<block_q8_0, block_q8_0, float>::gemm4xN<2>(long, long, long, long)
 13.34    366.40    53.82 115047805     0.00     0.00  void (anonymous namespace)::tinyBLAS<8, float __vector(8), float __vector(8), unsigned short, unsigned short, float>::gemm_b>  2.06    374.71     8.31     3482     2.39     2.39  ggml_compute_forward_glu
  2.03    382.92     8.21     4688     1.75     2.29  ggml_compute_forward_soft_max
  1.24    387.92     5.00 23187277     0.00     0.00  ggml_vec_dot_q8_0_q8_0
  0.63    390.45     2.53  1925309     0.00     0.00  ggml_vec_soft_max_f32

If I bypass the should_send logic, and just send_progress_response() when a batch is complete, I can set e.g. --batch-size 64 on the server command and get very reasonable progress behavior on a Raspberry Pi 5 running Gemma 4B. Which is what I'm after here.

If some other PR has done this better, I'm happy to go try one. This PR almost gets the behavior perfect.

[BradHutchings]

This probably doesn't qualify as a test case, but the video shows this code, with the change to logic of when to send progress that I described above, is working as it should. 19K tokens, batch size 64, Raspberry Pi 5, generic CPU code, Gemma 1B.

Mmojo.Progress.mp4

-Brad

[baonudesifeizhai]

This probably doesn't qualify as a test case, but the video shows this code, with the change to logic of when to send progress that I described above, is working as it should. 19K tokens, batch size 64, Raspberry Pi 5, generic CPU code, Gemma 1B.

Mmojo.Progress.mp4
-Brad

am just fixing it now

[ngxson]

server test is tools/server/tests/unit/test_chat_completion.py

not ctest

[ngxson]

Honestly by this point I'm spending more time reviewing this PR than just fixing it myself..

You clearly haven't even read the exiting code. pytest-compatible test is required.

[baonudesifeizhai]

Honestly by this point I'm spending more time reviewing this PR than just fixing it myself..

You clearly haven't even read the exiting code. pytest-compatible test is required.
sorry my bad ,wont happen again